A Multimodal Feature Selection Method for Remote Sensing Data Analysis Based on Double Graph Laplacian Diagonalization

When dealing with multivariate remotely sensed records collected by multiple sensors, an accurate selection of information at the data, feature, or decision level is instrumental in improving the scenes’ characterization. This will also enhance the system’s efficiency and provide more details on modeling the physical phenomena occurring on the Earth’s surface. In this article, we introduce a flexible and efficient method based on graph Laplacians for information selection at different levels of data fusion. The proposed approach combines data structure and information content to address the limitations of existing graph-Laplacian-based methods in dealing with heterogeneous datasets. Moreover, it adapts the selection to each homogenous area of the considered images according to their underlying properties. Experimental tests carried out on several multivariate remote sensing datasets show the consistency of the proposed approach

Impact of Noise Correlation on Multimodality

International audience—In this paper, we consider the problem of estimating an unknown random scalar observed by two modalities. We study two scenarios using mutual information and mean square error. In the first scenario, we consider that the noise correlation is known and examine its impact on the information content of two modalities. In the second scenario we quantify the information loss when the considered value of the noise correlation is wrong. It is shown that the noise correlation usually enhances the estimation accuracy and increases information. However, the performance declines if the noise correlation is misdefined, and the two modalities may jointly convey less information than one single modality

On the Exploitation of Heterophily in Graph-Based Multimodal Remote Sensing Data Analysis

Source at https://ceur-ws.org/.The field of Earth observation is dealing with increasingly large, multimodal data sets. An important processing step consists of providing these data sets with labels. However, standard label propagation algorithms cannot be applied to multimodal remote sensing data for two reasons. First, multimodal data is heterogeneous while classic label propagation algorithms assume a homogeneous network. Second, real-world data can show both homophily (’birds of a feather flock together’) and heterophily (’opposites attract’) during propagation, while standard algorithms only consider homophily. Both shortcomings are addressed in this work and the result is a graph-based label propagation algorithm for multimodal data that includes homophily and/or heterophily. Furthermore, the method is also able to transfer information between uni- and multimodal data. Experiments on the remote sensing data set of Houston, which contains a LiDAR and a hyperspectral image, show that our approach ties state-of-the-art methods for classification with an OA of 91.4%, while being more flexible and not constrained to a specific data set or a specific combination of modalities

SAR and Passive Microwave Fusion Scheme: A Test Case on Sentinel-1/AMSR-2 for Sea Ice Classification

The most common source of information about sea ice conditions is remote sensing data, especially images obtained from synthetic aperture radar (SAR) and passive microwave radiometers (PMR). Here we introduce an adaptive fusion scheme based on Graph Laplacians that allows us to retrieve the most relevant information from satellite images. In a first test case, we explore the potential of sea ice classification employing SAR and PMR separately and simultaneously, in order to evaluate the complementarity of both sensors and to assess the result of a combined use. Our test case illustrates the flexibility and efficiency of the proposed scheme and indicates an advantage of combining AMSR-2 89 GHz and Sentinel-1 data for sea ice mapping

Automatic Selection of Relevant Attributes for Multi-Sensor Remote Sensing Analysis: A Case Study on Sea Ice Classification

It is of considerable benefit to combine information obtained from different satellite sensors to achieve advanced and improved characterization of sea ice conditions. However, it is also true that not all the information is relevant. It may be redundant, corrupted, or unnecessary for the given task, hence decreasing the performance of the algorithms. Therefore, it is crucial to select an optimal set of image attributes which provides the relevant information content to enhance the efficiency and accuracy of the image interpretation and retrieval of geophysical parameters. Comprehensive studies have been focused on the analysis of relevant features for sea ice analysis obtained from different sensors, especially synthetic aperture radar. However, the outcomes of these studies are mostly data and application-dependent and can, therefore, rarely be generalized. In this article, we employ a feature selection method based on graph Laplacians, which is fully automatic and easy to implement. The proposed approach assesses relevant information on a global and local level using two metrics and selects relevant features for different regions of an image according to their physical characteristics and observation conditions. In the recent study, we investigate the effectiveness of this approach for sea ice classification, using different multi-sensor data combinations. Experiments show the advantage of applying multi-sensor data sets and demonstrate that the attributes selected by our method result in high classification accuracies. We demonstrate that our approach automatically considers varying technical, sensor-specific, environmental, and sea ice conditions by employing flexible and adaptive feature selection method as a pre-processing step

Interaction model and performance of multimodal signal processing

Bien que le traitement conjoint des mesures multimodales soit supposé conduire à de meilleures performances que celles obtenues en exploitant une seule modalité ou plusieurs modalités indépendamment, il existe des exemples en littérature qui prouvent que c'est pas toujours vrai. Dans cette thèse, nous analysons rigoureusement, en termes d'information mutuelle et d'erreur d'estimation, les différentes situations de l'analyse multimodale afin de déterminer les conditions conduisant à des performances optimales.Dans la première partie, nous considérons le cas simple de deux ou trois modalités, chacune étant associée à la mesure bruitée d'un signal, avec des liens entre modalités matérialisés par les corrélations entre les parties utiles du signal et par les corrélations les bruits. Nous montrons comment les performances obtenues sont améliorées avec l'exploitation des liens entre les modalités. Dans la seconde partie, nous étudions l'impact sur les performances d'erreurs sur les liens entre modalités. Nous montrons que ces fausses hypothèses dégradent les performances, qui peuvent alors devenir inférieure à celles atteintes avec une seule modalité.Dans le cas général, nous modélisons les multiples modalités comme un canal gaussien bruité. Nous étendons alors des résultats de la littérature en considérant l'impact d'erreurs sur les densités de probabilité du signal et du bruit sur l'information transmise par le canal. Nous analysons ensuite cette relation dans la cas d'un modèle simple de deux modalités. Nos résultats montrent en particulier le fait inattendu qu'une double inadéquation du bruit et du signal peuvent parfois se compenser et ainsi conduire à de très bonnes performances.The joint processing of multimodal measurements is supposed to lead to better performances than those obtained using a single modality or several modalities independently. However, in literature, there are examples that show that is not always true. In this thesis, we analyze, in terms of mutual information and estimation error, the different situations of multimodal analysis in order to determine the conditions to achieve the optimal performances.In the first part, we consider the simple case of two or three modalities, each associated with noisy measurement of a signal. These modalities are linked through the correlations between the useful parts of the signal and the correlations between the noises. We show that the performances are improved if the links between the modalities are exploited. In the second part, we study the impact on performance of wrong links between modalities. We show that these false assumptions decline the performance, which can become lower than the performance achieved using a single modality.In the general case, we model the multiple modalities as a noisy Gaussian channel. We then extend literature results by considering the impact of the errors on signal and noise probability densities on the information transmitted by the channel. We then analyze this relationship in the case of a simple model of two modalities. Our results show in particular the unexpected fact that a double mismatch of the noise and the signal can sometimes compensate for each other, and thus lead to very good performances

Modèle d'interaction et performances du traitement du signal multimodal

The joint processing of multimodal measurements is supposed to lead to better performances than those obtained using a single modality or several modalities independently. However, in literature, there are examples that show that is not always true. In this thesis, we analyze, in terms of mutual information and estimation error, the different situations of multimodal analysis in order to determine the conditions to achieve the optimal performances.In the first part, we consider the simple case of two or three modalities, each associated with noisy measurement of a signal. These modalities are linked through the correlations between the useful parts of the signal and the correlations between the noises. We show that the performances are improved if the links between the modalities are exploited. In the second part, we study the impact on performance of wrong links between modalities. We show that these false assumptions decline the performance, which can become lower than the performance achieved using a single modality.In the general case, we model the multiple modalities as a noisy Gaussian channel. We then extend literature results by considering the impact of the errors on signal and noise probability densities on the information transmitted by the channel. We then analyze this relationship in the case of a simple model of two modalities. Our results show in particular the unexpected fact that a double mismatch of the noise and the signal can sometimes compensate for each other, and thus lead to very good performances.Bien que le traitement conjoint des mesures multimodales soit supposé conduire à de meilleures performances que celles obtenues en exploitant une seule modalité ou plusieurs modalités indépendamment, il existe des exemples en littérature qui prouvent que c'est pas toujours vrai. Dans cette thèse, nous analysons rigoureusement, en termes d'information mutuelle et d'erreur d'estimation, les différentes situations de l'analyse multimodale afin de déterminer les conditions conduisant à des performances optimales.Dans la première partie, nous considérons le cas simple de deux ou trois modalités, chacune étant associée à la mesure bruitée d'un signal, avec des liens entre modalités matérialisés par les corrélations entre les parties utiles du signal et par les corrélations les bruits. Nous montrons comment les performances obtenues sont améliorées avec l'exploitation des liens entre les modalités. Dans la seconde partie, nous étudions l'impact sur les performances d'erreurs sur les liens entre modalités. Nous montrons que ces fausses hypothèses dégradent les performances, qui peuvent alors devenir inférieure à celles atteintes avec une seule modalité.Dans le cas général, nous modélisons les multiples modalités comme un canal gaussien bruité. Nous étendons alors des résultats de la littérature en considérant l'impact d'erreurs sur les densités de probabilité du signal et du bruit sur l'information transmise par le canal. Nous analysons ensuite cette relation dans la cas d'un modèle simple de deux modalités. Nos résultats montrent en particulier le fait inattendu qu'une double inadéquation du bruit et du signal peuvent parfois se compenser et ainsi conduire à de très bonnes performances

Selecting principal attributes in multimodal remote sensing for sea ice characterization

Automatic ice charting cannot be achieved using only SAR modalities. It is fundamental to combine information from other remote sensors with different characteristics for more reliable sea ice characterization. In this paper, we employ principal feature analysis (PFA) to select significant information from multimodal remote sensing data. PFA is a simple yet very effective approach that can be applied to several types of data without loss of physical interpretability. Considering that different homogeneous regions require different types of information, we perform the selection patch-wise. Accordingly, by exploiting the spatial information, we increase the robustness and accuracy of PFA

On Enhanced Ensemble Learning for Multimodal Remote Sensing Data Analysis by Capacity Optimization

International audienceMultimodal remote sensing data analysis can strongly improve the characterization of physical phenomena on Earth's surface. Nonetheless, nonidealities and estimation imperfections between records and investigation models can limit its information extraction ability. Ensemble learning could be used to tackle these issues. Combining the information acquired by multiple weak classifiers can prevent the analysis of large scale heterogeneous datasets from being affected by overfitting and biasing. In this paper, we introduce an enhanced ensemble learning scheme where the information acquired by the weak classifiers is combined to optimize the maximum information extraction for the given system at a decision level. Using an asymptotic information theory-based approach, we define the capacity index associated with the maximum accuracy that can be achieved under optimal conditions for multimodal analysis. By selecting the decisions delivered by the different classifiers according to the capacity optimization, the performance of the ensemble learning scheme will be maximized